{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "# Admin Startup\n",
    "\n",
    "The purpose of this notebook is to show how to start an admin client to operate an FL experiment with a server and at least one client started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "## Prerequisites\n",
    "- The [Startup notebook](Startup.ipynb) has been run successfully.\n",
    "- A server and at least one client has been started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the Config Folder\n",
    "\n",
    "The config folder should be in the `transfer/` directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['spleen_example']"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from IPython.display import HTML\n",
    "from multiprocessing import Process\n",
    "\n",
    "workspace = \"demo_workspace/\"\n",
    "admin_name = \"admin@nvidia.com\"\n",
    "\n",
    "transfer_path = os.path.join(workspace, admin_name, \"transfer\")\n",
    "os.listdir(transfer_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edit Docker Script\n",
    "\n",
    "Before starting the docker script, you need to edit it to ensure that the environments (such as dataset, GPUs, memory, ...) meet your requirement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Please click here to check the admin@nvidia.com script](demo_workspace/admin@nvidia.com/startup/docker.sh)\n",
    "\n",
    "For this experiment, please modify the script in order to use the host network:\n",
    "\n",
    "```\n",
    "#!/usr/bin/env bash\n",
    "DIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" >/dev/null 2>&1 && pwd )\"\n",
    "# docker run script for FL admin\n",
    "# to use host network, use line below\n",
    "NETARG=\"--net=host\"\n",
    "# Admin clients do not need to open ports, so the following line is not needed.\n",
    "#NETARG=\"-p 8003:8003\"\n",
    "DOCKER_IMAGE=monai_nvflare:latest\n",
    "echo \"Starting docker with $DOCKER_IMAGE\"\n",
    "docker run --rm -it --name=fladmin -v $DIR/..:/workspace/ -w /workspace/ --ipc=host $NETARG $DOCKER_IMAGE /bin/bash\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Start Admin Docker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running cmd demo_workspace/admin@nvidia.com/startup/docker.sh\n",
      "Starting docker with monai_nvflare:latest\n",
      "\n",
      "=============\n",
      "== PyTorch ==\n",
      "=============\n",
      "\n",
      "NVIDIA Release 21.08 (build 26011915)\n",
      "PyTorch Version 1.10.0a0+3fd9dcf\n",
      "\n",
      "Container image Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "Copyright (c) 2014-2021 Facebook Inc.\n",
      "Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)\n",
      "Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)\n",
      "Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)\n",
      "Copyright (c) 2011-2013 NYU                      (Clement Farabet)\n",
      "Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)\n",
      "Copyright (c) 2006      Idiap Research Institute (Samy Bengio)\n",
      "Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)\n",
      "Copyright (c) 2015      Google Inc.\n",
      "Copyright (c) 2015      Yangqing Jia\n",
      "Copyright (c) 2013-2016 The Caffe contributors\n",
      "All rights reserved.\n",
      "\n",
      "NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "This container image and its contents are governed by the NVIDIA Deep Learning Container License.\n",
      "By pulling and using the container, you accept the terms and conditions of this license:\n",
      "https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license\n",
      "\n",
      "NOTE: MOFED driver for multi-node communication was not detected.\n",
      "      Multi-node communication performance may be reduced.\n",
      "\n",
      "\u001b]0;root@sys: /workspace\u0007root@sys:/workspace# "
     ]
    }
   ],
   "source": [
    "admin_startup_path = os.path.join(workspace, admin_name, \"startup\")\n",
    "cmd = admin_startup_path + \"/docker.sh\"\n",
    "\n",
    "\n",
    "def run_admin():\n",
    "    cmd = admin_startup_path + \"/docker.sh\"\n",
    "    print(\"running cmd \" + cmd)\n",
    "    !$cmd\n",
    "\n",
    "\n",
    "p1 = Process(target=run_admin)\n",
    "\n",
    "p1.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Started Containers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS              PORTS     NAMES\n",
      "3de0f6690d8e   monai_nvflare:latest   \"/usr/local/bin/nvid…\"   6 seconds ago        Up 6 seconds                  fladmin\n",
      "96c8f9406303   monai_nvflare:latest   \"/usr/local/bin/nvid…\"   52 seconds ago       Up 52 seconds                 org1-a\n",
      "6b5d9eefc699   monai_nvflare:latest   \"/usr/local/bin/nvid…\"   52 seconds ago       Up 52 seconds                 org1-b\n",
      "08492335d003   monai_nvflare:latest   \"/usr/local/bin/nvid…\"   About a minute ago   Up About a minute             flserver\n"
     ]
    }
   ],
   "source": [
    "!docker ps -a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Admin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start an admin, you should:\n",
    "\n",
    "- open a terminal and enter the container named `fladmin`.\n",
    "- run `fl_admin.sh` under `startup/`.\n",
    "- input admin name `admin@nvidia.com`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href=\"\", data-commandlinker-command=\"terminal:create-new\"> Open a new terminal</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML('<a href=\"\", data-commandlinker-command=\"terminal:create-new\"> Open a new terminal</a>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can accomplish these steps by running:\n",
    "\n",
    "```\n",
    "docker exec -it fladmin bash\n",
    "cd startup/\n",
    "bash fl_admin.sh\n",
    "admin@nvidia.com\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare for the experiment\n",
    "\n",
    "You need to execute the following steps to prepare for the experiment:\n",
    "\n",
    "- upload pipeline config folder\n",
    "- set FL training number\n",
    "- deploy the folder to client(s) and server\n",
    "\n",
    "The commands can be:\n",
    "```\n",
    "upload_app spleen_example\n",
    "set_run_number 1\n",
    "deploy_app spleen_example server\n",
    "deploy_app spleen_example client\n",
    "```\n",
    "\n",
    "Now, let's check if the folder has been distributed into the server and all client(s):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "config files on server: ['mmar_server', 'mmar_org1-a', 'mmar_org1-b']\n",
      " \n",
      "config files on org1-a: ['mmar_org1-a']\n",
      " \n",
      "config files on org1-b: ['mmar_org1-b']\n",
      " \n"
     ]
    }
   ],
   "source": [
    "run_file = 'run_1'\n",
    "\n",
    "for name in ['server', 'org1-a', 'org1-b']:\n",
    "    path = os.path.join(workspace, name, run_file)\n",
    "    if os.path.exists(path):\n",
    "        print(\"config files on {}: {}\".\n",
    "              format(name, os.listdir(path)))\n",
    "        print(\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example prepares two different data list files: `dataset_part1.json` and `dataset_part2.json`, and they have the same validation set and totally different training set. The default file used in `spleen_example/config_train.json` is `dataset_part1.json`. Therefore, if you want to let two clients train on different data, you can switch to use `dataset_part2.json` for `org1-b`.\n",
    "\n",
    "[Link to org1-a config](demo_workspace/org1-a/run_1/mmar_org1-a/config/config_train.json)\n",
    "\n",
    "[Link to org1-b config](demo_workspace/org1-b/run_1/mmar_org1-b/config/config_train.json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can start training with:\n",
    "```\n",
    "start_app server\n",
    "start_app client\n",
    "```\n",
    "You can check the status by running:\n",
    "```\n",
    "check_status server\n",
    "check_status client\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop Training\n",
    "\n",
    "You can stop training for the server and/or client(s) by running:\n",
    "```\n",
    "abort client\n",
    "abort server\n",
    "```\n",
    "If you only want to stop org1-b, you can use:\n",
    "```\n",
    "abort client org1-b\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Training (multi-gpus)\n",
    "\n",
    "If you would like to use multiple gpus, you can start training with n gpus (where n is the number of gpus) by running:\n",
    "```\n",
    "start_mgpu client n\n",
    "```\n",
    "\n",
    "The default `multi_gpu` flag is `False` in `spleen_example/config/config_train.json`, if you need to use multiple gpus, you have to change it before the `Startup` step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shutdown\n",
    "\n",
    "You can shutdown the experiment for the server and/or client(s) by running:\n",
    "\n",
    "`shutdown client` and/or `shutdown server`\n",
    "\n",
    "If you only want to shutdown a specific client, you can specify the client in the command like follows:\n",
    "```\n",
    "shutdown client fl_clients_2\n",
    "```\n",
    "This command will kill the client/server connection, and this command will need input of the admin name for confirmation. If you need to shutdown the server, all active clients need to be shutdown first.\n",
    "\n",
    "After the shutdown commands, you are safe to shutdown `Startup.ipynb` to stop all containers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Commands\n",
    "\n",
    "Please type `?` to learn more about all commands."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next Steps\n",
    "\n",
    "You have now started the admin client and learnt the commands to control your FL experiment. You're now ready to create your own FL experiment!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "stem_cell": {
   "cell_type": "raw",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
